1. Field of the Invention
The present invention relates to devices and methods to increase the effective resolution of an array of spaced-apart light-sensitive transducers used to record an image in a digital imaging device.
2. Description of the Prior Art
Devices that record a visual scene by using an array of photoelectric sensors are well known in the art. Such a device configured as a scanner may be used to record digitally an existing transparency or print, or, in the form of a camera, may record a scene. The photoelectric-sensor array is typically a charge-coupled device (CCD) that outputs analog information that, in turn, is commonly digitized through the use of an analog-to-digital converter (ADC) and stored as bits of data in computer memory or other form of electronic storage. For this discussion we will refer to camera-type devices utilizing CCD array sensors, though the principles laid out here apply equally well to cameras constructed with other types of photoelectric sensors.
Often the CCD is the single most expensive part of the digital camera. The quality of the recorded image is related in part to the total number of sensing elements (pixels) that are present in the CCD. The greater this number, the higher the quality of the resulting image. In fact, it is the number of pixels that the camera can capture that will often distinguish one camera from another in the marketplace. Thus any device or method that can increase the effective pixel count of a CCD without causing the actual count to increase is of interest to camera manufacturers since such a device can achieve higher performance with minimal cost increase.
Professional digital cameras used in the studio tend to fall into two categories: (1) those that can record action scenes, but sacrifice color accuracy and suffer from color aliasing; (2) those that can only record still-life scenes, but have higher accuracy and resolution enhancement. Since most studios must photograph both types of scenes, most studios are compelled to own both types of cameras at a considerable financial burden. A single camera that can function in both modes would be very desirable.
Definition of Terms
An xe2x80x9carray sensorxe2x80x9d is a sensor containing an array of tiled sensor pixels, i.e., the sensor pixels are arranged in contiguous horizontal rows and vertical rows. A xe2x80x9csensor pixelxe2x80x9d is defined as the smallest unit of area on the sensor that can be tiled to create an array. By this definition, a sensor pixel may include both light-sensitive and non-light-sensitive regions. Referring to FIG. 1, array 10 is comprised of many instances of pixel 11, pixel 11 being further comprised of light-sensitive region 12 and non-light-sensitive region 14. xe2x80x9cSensor pixel pitchxe2x80x9d in either the X or Y direction is equal to the dimension of sensor pixel 10 in the corresponding direction. (Alternatively, it can be described as the center-to-center distance between pixels in the stated direction.) xe2x80x9cAperture ratioxe2x80x9d is defined as the area of light-sensitive region 12 divided by the total pixel area (the sum of the area of non-light-sensitive region 14 and light-sensitive region 12 ). Aperture ratio can be decomposed into an X component and a Y component. FIG. 1 represents an array with an aperture ratio of about 25% (50% in X and 50% in Y). FIG. 2 represents an array with an aperture ratio of about 50% (50% in X and about 100% in Y). FIG. 3 represents an array with a nearly 100% aperture ratio. The term xe2x80x9cresolutionxe2x80x9d refers to a total number of pixels being fixed. Resolution is a measure of information. xe2x80x9cNative resolutionxe2x80x9d refers to the actual number of pixels in the imager. xe2x80x9cEffective resolutionxe2x80x9d produced from some resolution-enhancement technique is defined to be numerically equal to the native resolution that would be otherwise needed to create the same amount and quality of pixel information without said technique.
The terms xe2x80x9cimage pixelxe2x80x9d and xe2x80x9csensor pixelxe2x80x9d distinguish the image realm from the sensor realm. An image is typically comprised of image pixels that are computed from sensor pixel data using a correspondence that is often, but not necessarily one-to-one.
Many devices and methods have been proposed to make the effective sensor resolution higher than the native sensor resolution. In general, these approaches can be divided into three main categories.
Three Approaches to Increase Sensor Resolution.
The most straightforward approach is the tiling approach, which is to move, between successive exposures, the entire sensing array once or multiple times a distance equal to the width and/or height of the whole array and tiling the resulting image pixels together. The effective sensor resolution is higher than the native resolution by the number of such moves that occur in the course of producing the net image of such tiles.
The tiling approach has several disadvantages. First, it is difficult to implement, since the translation device must rapidly and with sub-pixel pitch accuracy displace the array a distance equal to its full width and/or height with repeatable results. Since the typical imager will have thousands of pixels in each dimension, the positioning accuracy must be, say, one part in ten thousand or better to get good tiling. Also, the imaging array must not be displaced in a direction normal to the focal plane or a deleterious focus shift will occur. Finally, such an approach has the effect of making the angle of view dependent on the size of the tiling for a given focal length lens. This is a very awkward situation since a non-tiled image does not yield a preview of a final higher resolution image but rather only a section of the higher resolution image. Thus the whole tiling procedure must be completed before viewing a high resolution image is possible. If the photographer completes his or her image at a low resolution (say a single tile for example) and is satisfied with the aesthetics, but decides the image lacks sufficient resolution, then the new exposure must be tiled demanding that the photographer change lenses and recompose. This change of lens is slow, and usually requires re-focusing and resetting the aperture. A zoom lens can be used to overcome the lens substitution problem, but not the re-composition problem, and zoom lenses usually possess a lower resolving power than the same fixed focal length lens. FIGS. 5A and 5B illustrate the tiling approach. FIG. 5A shows a 3xc3x973 array with a native resolution f9 and angle of view theta; FIG. 5B shows the 3xc3x973 array tiled 3 times, with a native resolution of 9 and an enhanced resolution of 27. The angle of view is (2)xc3x97theta.
It would be much more desirable to maintain the same angle of view independent of effective resolution. This would allow a quick preview for aesthetic composition for example.
The second approach is the interstitial approach. This approach depends on the fact that the aperture ratio of some imagers is or can be made to be equal to or less than 50% in at least one (X or Y) direction, i.e., part of each pixel is non-sensitive. For example, an interline transfer CCD has columns of non-sensitive regions interleaved with the sensitive regions. Such imagers are common in video cameras. A way to increase the effective sensor resolution with such imagers is to shift light-sensitive regions into the non-sensitive regions. By shifting such a xe2x80x9csparsexe2x80x9d CCD array by an amount sufficient to re-position the light sensitive areas to a new position previously occupied entirely by non-sensitive areas, a new array of image data is produced that interleaves with the native data, thus increasing the effective sensor pixel count. The success of this approach depends in part on the fact that the native and shifted data sets are not convoluted because there is no spatial overlap between the unshifted and shifted light-sensitive positions.
Many such shifting methods exist and all depend on the existence of insensitive regions of the array into which the sensitive areas may be shifted. Torok et al. (U.S. Pat. No. 5,489,994; 1996) teaches a modification of an existing CCD design by adding said insensitive regions that otherwise would not exist, thereby enabling the CCD to be used in such an interstitial resolution-enhancement mode. FIG. 6 illustrates the use of the interstitial resolution-enhancement technique. In FIG. 6A 3xc3x973 array is shown that has a 25% aperture ratio. Referring to FIG. 6B, four exposures are taken, a first exposure, then a second where the array is displaced in the X direction by the width of the light-sensitive region 12, then a third after a displacement in the negative Y direction and finally a fourth after a negative X displacement. The resulting 36 member virtual array is illustrated in FIG.
While the interstitial shifting technique does increase the effective resolution of the array it suffers from several drawbacks. First, the most faithful reproduction of an image projected onto an array of sensing elements is achieved when the aperture ratio nears 100%. As the aperture ratio decreases below 100%, pixel sampling error becomes more and more pronounced. Thus if one intends to have a low resolution mode as well as a high resolution mode employing interstitial shifting, the quality of the low resolution mode will be compromised by sampling error. Another drawback of the low aperture ratio interstitial shifting method is compromised pixel sensitivity. The light sensitivity of a given pixel in a CCD as measured by the number of electrons produced per unit of illumination is linearly related to the size of the light-sensitive area of the pixel all other factors held constant. Since a more sensitive pixel is a general design goal by virtue of improved signal-to-noise performance and because a more sensitive imager is generally desirable for well known reasons, CCD""s with small aperture ratios are not as desirable for still photography as those with aperture ratios approaching 100%. A further drawback of the low-aperture-ratio interstitial-shifting method is that the degree of resolution-enhancement that can be achieved depends on the aperture ratio. For example, if the aperture ratio is 50% laterally and 100% vertically, then only a resolution doubling can be achieved laterally, and no enhancement is possible vertically. As another example, if the artificial apertures as taught by Torok et al. are contrived to create a 50% aperture ratio both laterally and vertically (25% total aperture ratio) as illustrated in FIG. 6, then a resolution doubling both laterally and vertically is possible, but not say, tripling or quadrupling.
The third approach is the overlapped shifting approach, which achieves higher resolution by shifting the sensing array and allowing overlap of the pixels. While there is much in the prior art describing approaches that shift arrays using the interstitial method, there is very little describing the overlapped shifting, or sub-pixel-shifting, approach. Hersh (U.S. Pat. No. 4,992,878; 1991) teaches how to increase the effective pixel count by moving a color-filtered array a distance less than a xe2x80x9cpixelxe2x80x9d width between exposures of an invariant scene and extracting the resulting higher resolution information. However, the Hersh invention considers a xe2x80x98pixelxe2x80x99 as comprising a cluster of four neighboring but distinct color-filtered sensing elements: clear, yellow filtered, cyan filtered, and green filtered. By considering a pixel to be a composite of four sensing elements, several serious disadvantages are introduced. First, the native resolution becomes one-fourth what it would be if the individual sensing elements were considered to be independent pixels, thus partially defeating the very purpose of the invention.
Also, the cited prior art fails to address the problem of color aliasing. Color aliasing problems potentially exist for all camera configurations that use local color filtering of the sensing matrix unless specific anti-aliasing techniques are employed. Color aliasing exists when using a color-filtered array (CFA) because at any given pixel, two out of the three primary colors (in, for example, an R-G-B- system) have not been sampled. The missing data must be interpolated from neighboring pixels, and the interpolation can only approximate the true data. Color aliasing is most troublesome when the spatial frequency of the image formed on the CCD (or other his type of imager) meets or exceeds the Nyquist frequency of the imager (xc2xd the sampling spatial frequency). Furthermore, color aliasing reduces the usable resolution of a CFA below what would be expected from the size of the array. For example, a monochrome array capturing a monochrome image will faithfully record higher spatial frequency data than will the same sized CFA recording the same monochrome image.
FIG. 7 illustrates the color aliasing phenomenon. FIG. 7A shows a typical CFA in the Bayer pattern: 50% green imaging sites, 25% blue and 25% red. A monochrome image is projected onto the CFA in the pattern shown in FIG. 7B. As can be visualized, the pattern is a simple bright gray column (intensity=50) in a field of dark gray (intensity=10). Monochrome is defined here to mean equal parts of red, green, and blue light components. It is clear from FIG. 7C that only a fraction of the actual color component intensities are measured by the CFA. The CFA has trouble with this image since the spatial frequency of the image is higher than the Nyquist frequency of the CFA. A simple nearest neighbor interpolation to recover the missing data is illustrated in FIG. 7D. As can been seen in FIG. 7D, there is significant false color generation on either side of the bright column, and the spatial fidelity is reduced as well. Of course other interpolation schemes are possible, but no interpolation scheme can overcome the fact that the CFA records only a fraction of all the information needed to faithfully record a color image at a spatial frequency approaching the Nyquist frequency of the CFA. The problem can be described in terms of information theory. The total information needed to record a color image that fills an arbitrary n*m array is five dimensions of data: the two spatial dimensions (X and Y) and the three color dimensions (red, green, and blue in the case of the RGB system). Thus for the arbitrary n*m array, a data set totaling n*m*3 pieces of data must be assembled to faithfully record a color image at the Nyquist frequency. Clearly the CFA records just 60% of the needed data space, and a tradeoff must be made between color fidelity and spatial fidelity. Each piece of data typically contains 8 to 14 bits of data, the bit depth being determined by the bit resolution of the ADC. Higher bit depths allow more faithful tonal reproduction, lower bit depths in the extreme can lead to overly quantized tonal reproduction (xe2x80x9cposterizationxe2x80x9d).
Color aliasing can be attacked by holding the spatial frequency of the image below the Nyquist frequency of the imager (by using a form of blurring filter, for example), but the tradeoff expressed above forces a resolution loss. Resolution-enhancement loses value now since there is little additional image information available anyway. If no blurring filter is used, then the problem of color aliasing deflates the resolution-enhancement achieved and one is left with the problem of false color.
What is needed is a resolution-enhancement method that allows the use of high-aperture-ratio sensing arrays, that enables resolution-enhancement, independent of the angle of view, and that provides multiples of higher resolution in a plurality of resolution-enhancement modes that are variable in horizontal and vertical axes. What is further needed is such a method that suppresses color aliasing without requiring macro movement of the sensing array. What is yet further needed is such a method that enables use of a single camera for both single-exposure and multiple-exposure work.
It is an object of the present invention to provide a color still-image pickup method that allows the use of high-aperture-ratio sensing arrays that enables resolution-enhancement, independent of the angle of view. It is a further object of the present invention to provide such a method that provides multiples of higher resolution in a plurality of resolution-enhancement modes that are variable in the horizontal and vertical axes. It is a yet further object of the present invention to provide such a method that suppresses color-aliasing in a multi-exposure native-resolution mode. It is a still further object of the present invention to provide such a method that enables use of a single camera for work in both single-exposure and multiple-exposure modes.
The objectives are achieved by providing a resolution-enhancement method of taking still-image exposures in a multiple-exposure technique, using a sub-pixel overlap in conjunction with a whole-pixel shift In the Preferred Embodiment, only two exposures per degree of resolution-enhancement are required, as compared to, say, the three exposures required by a global filtered tiling method. Because the Preferred Embodiment uses a CFA sensing device (CCD), a single exposure at native resolution is sufficient to provide a full color image to the same scale as any subsequent enhanced resolution without requiring any change of lens.
In an alternate embodiment, an unfiltered (monochrome) CCD is used in conjunction with global color filtering to suppress color aliasing. In the alternate embodiment all of the advantages of the Preferred Embodiment are preserved, except that three exposures are required at native resolution to form a color preview.